Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type.

This work describes experiments done using pretrained word embedding model representations, and fMRI and MEG representations when reading the same text. Bringing these together is original and an interesting avenue of research, yet I have doubts about the significance, clarity and quality of this work. There are several points where references would have been needed to refer to prior work, or to back up some claim (e.g. The paper furthermore has a significant part of its material deferred into the appendix, including parts that are crucial to understand the experiment. From the main paper alone it is for example unclear which metrics are used when evaluating the fitted linear models, and even information as basic as whether the task is a regression or a classification task.

This is an important goal, because humans have neural networks that can make sense of language, and there may be a lot that the machine learning community can learn from understanding these exemplary language processors better. Indeed, a teleological explanation of how human or artificial neural-networks process language would be an enormous breakthrough in language science. The reviewers are positive about this paper and therefore I support them in recommending acceptance. To add to the positive aspects that they point out, I have a few concerns about the framing, which I list below in case they can improve the camera ready submission: 1) The abstract states that: "it is still unclear what the representations learned by these networks correspond to". It seems to me that this paper does not really answer the question it poses in the first line of the abstract.

Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type.

In early 2011, Ken Jennings looked like humanity's last hope. Watson, an artificial intelligence created by the tech giant IBM, had picked off lesser Jeopardy players before the show's all-time champ entered a three-day exhibition match. At the end of the first game, Watson--a machine the size of 10 refrigerators--had Jennings on the ropes, leading $35,734 to $4,800. On day three, Watson finished the job. "I for one welcome our new computer overlords," Jennings wrote on his video screen during Final Jeopardy. Watson was better than any previous AI at addressing a problem that had long stumped researchers: How do you get a computer to precisely understand a clue posed in idiomatic English and then spit out the correct answer (or, as in Jeopardy, the right question)?

The question will be simple but perpetual: Person or machine? Every encounter with language, other than in the flesh, will now bring with it that small, consuming test. For some--teachers, professors, journalists--the question of humanity will be urgent and essential. For those who operate in the large bureaucratic apparatus of boilerplate--copywriters, lawyers, advertisers, political strategists--the question will be irrelevant except as a matter of efficiency. How will they use new artificial-intelligence technology to accelerate the production of language that was already mostly automatic? For everyone, the question will now hover, quotidian and cosmic, over words wherever you find them: Who's there?

Suppose you are a professor of pedagogy, and you assign an essay on learning styles. The construct of "learning styles" is problematic because it fails to account for the processes through which learning styles are shaped. Some students might develop a particular learning style because they have had particular experiences. Others might develop a particular learning style by trying to accommodate to a learning environment that was not well suited to their learning needs. Ultimately, we need to understand the interactions among learning styles and environmental and personal factors, and how these shape how we learn and the kinds of learning we experience. And how would your grade change if you knew a human student hadn't written it at all?

In the back room of an old and graying building in the northernmost region of New Zealand, one of the most advanced computers for artificial intelligence is helping to redefine the technology's future. Te Hiku Media, a nonprofit Māori radio station run by life partners Peter-Lucas Jones and Keoni Mahelona, bought the machine at a 50% discount to train its own algorithms for natural-language processing. Mahelona, a native Hawaiian who settled in New Zealand after falling in love with the country, chuckles at the irony of the situation. "The computer is just sitting on a rack in Kaitaia, of all places--a derelict rural town with high poverty and a large Indigenous population. I guess we're a bit under the radar," he says. As a nonprofit journalism organization, we depend on your support to fund coverage of Indigenous issues and communities. Donate any amount today to become a Pulitzer Center Champion and receive exclusive benefits!

An enormous figure looms over scientists searching for new drugs: the estimated US$2.6-billion price tag of developing a treatment. A lot of that effectively goes down the drain, because it includes money spent on the nine out of ten candidate therapies that fail somewhere between phase I trials and regulatory approval. Few people in the field doubt the need to do things differently. Leading biopharmaceutical companies believe a solution is at hand. Pfizer is using IBM Watson, a system that uses machine learning, to power its search for immuno-oncology drugs.

This concise, project-driven guide to deep learning takes readers through a series of program-writing tasks that introduce them to the use of deep learning in such areas of artificial intelligence as computer vision, natural-language processing, and reinforcement learning. The author, a longtime artificial intelligence researcher specializing in natural-language processing, covers feed-forward neural nets, convolutional neural nets, word embeddings, recurrent neural nets, sequence-to-sequence learning, deep reinforcement learning, unsupervised models, and other fundamental concepts and techniques. Students and practitioners learn the basics of deep learning by working through programs in Tensorflow, an open-source machine learning framework. "I find I learn computer science material best by sitting down and writing programs," the author writes, and the book reflects this approach. Each chapter includes a programming project, exercises, and references for further reading.